Selecting one of a plurality of adapters to use to transmit a packet

ABSTRACT

Provided are a method, system, and program for selecting one of a plurality of adapters to use to transmit a packet. A packet is generated by a protocol driver having connection information in a packet header. A determination is made as to whether there is one value in a table for the connection information. The packet and the determined value are forwarded to a network driver in response to determining that the table has one value for the connection information. The network driver uses the forwarded value to determine one of a plurality of adapters to use to transmit the packet over a network.

BACKGROUND

A host computer may have multiple network cards to connect the system toa network. In current systems, the host may include a virtual networkinterface driver capable of teaming. A teaming program executes tointerface between the host operating system and the network cards. Forhosts including multiple adapters, the device driver would select one ofthe network cards to transmit a packet. The device driver executing inthe host may execute a load balancing algorithm to select a particularnetwork card to transmit the packet.

In certain systems, the virtual network interface may maintain a hashtable having entries associated with different adapters. The virtualnetwork interface applies a hash function to the connection information,such as the destination and source addresses and ports, in each outboundpacket to produce a hash value that is associated with one entry in thehash table. The hash table entry is associated with an adapter. Thistechnique ensures that packets for one connection always use the sameadapter because the connection information provides the same hash valuethat maps to one hash table entry corresponding to an adapter to whichthe packet is forwarded for transmission. For instance, if the hashtable has 256 entries, then these 256 entries can map to any of theavailable adapters. Multiple connections may hash to the same hash valueand, thus, the same adapter.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a computing environment in which embodiments areimplemented.

FIG. 2 illustrates an embodiment of fields in a packet.

FIG. 3 illustrates an embodiment of control information sent with apacket.

FIGS. 4 and 5 illustrate an embodiment of operations to select anadapter to use to transmit a packet.

DETAILED DESCRIPTION

In the following description, reference is made to the accompanyingdrawings which form a part hereof and which illustrate severalembodiments. It is understood that other embodiments may be utilized andstructural and operational changes may be made without departing fromthe scope of the embodiments.

FIG. 1 illustrates a computing system 2 having at least one centralprocessing unit (CPU) 4 and a memory 6, such as a volatile memory, inwhich programs execute. The CPU 4 executes in the memory 6 one or moreapplication programs 8, an operating system 10, and one or more protocoldrivers 12 a, 12 b (two are shown) that generate packets to transmitusing different network transmission protocols, such as the TransmissionControl Protocol (TCP)/Internet Protocol (IP), the Internet PacketExchange (IPX) protocol, etc. Details of the TCP/IP protocol arediscussed in the publication “A TCP/IP Tutorial”, Network Working GroupRequest for Comments No. 1180 (1991) and the IPX protocol is describedin the publication “Novell IPX Over Various WAN Media (IPXWAN)”, NetworkWorking Group Request for Comments No. 1362 (1992). The protocol drivers12 a, 12 b forward packets to a virtual network interface 14, alsoexecuting in memory 6, that selects one of a plurality of networkadapters 16 a, 16 b, 16 c to use to transmit a packet received from theprotocol drivers 12 a, 12 b. The virtual network interface 14 transmitspackets to the network adapters 16 a, 16 b, 16 c over a bus 18.

In one embodiment, the protocol drivers 12 a, 12 b may perform many ofthe protocol and packet management operations. In an alternativeembodiment, the protocol drivers 12 a . . . 12 b may comprise a TCP/IPOffload Engine (TOE) driver, where most of the packet management andprotocol operations are performed in TOE circuitry in the adapters 16 a,16 b . . . 16 c and the protocol driver maintains connection stateinformation and performs a more limited set of operations, includingdetermining hash values as described below.

The adapters 16 a, 16 b, 16 c may be mounted on the system 2motherboard, such as a Local Area Network (LAN) on Motherboardimplementation, or alternatively be implemented in an expansion cardthat is inserted in a slot on the system 2 motherboard. The bus 18 maycomprise bus interfaces known in the art, such as the PeripheralComponent Interconnect (PCI) bus described in the publication“Conventional PCI 2.3”, published by the PCI-SIG.

One or more of the protocol drivers 12 a, 12 b may include a hashfunction 20 to generate a hash value from connection information in apacket identifying an entry in an adapter table 22 that is associatedwith one of the adapters 16 a, 16 b, 16 c. The connection information towhich the hash value applies may comprise some of the source anddestination information in the packet generated by the protocol driver,such as the source address, destination address, source port, and/ordestination port. Some protocol drivers, e.g., 12 b, may not include ahash function 20. Protocol drivers 12 a, 12 b that include a hashfunction 20 may maintain a connection hash table 24 having entries thatprovide an association of connection information for packets and a hashvalue so that the hash value corresponding to a connection may bedetermined by looking up the value in the connection hash table 24. Bylooking-up the hash value in the connection hash table 24, the systemavoids consuming computational resources to execute the hash function 20to recalculate the hash value from the connection information.

The virtual network interface 14 may also include a hash function 26 tocalculate the hash value from the connection information in a packetfrom protocol drivers, e.g., 12 b, that do not include a hash function20 and connection hash table 24 to determine the hash value for aconnection. The hash function 26 may utilize the same algorithm as thehash function 20. Further, different hash operations may apply to theconnection information for different protocol drivers.

FIG. 2 illustrates information maintained in a packet 50 generated by aprotocol driver 12 a, 12 b, including a source port 52, a destinationport 54, a source address 56, a destination address 58, and packet data60. The fields 50-60 may be included in the packet header.

FIG. 3 illustrates control information 70 the protocol driver 12 a, 12 bforwards to the virtual network interface 14. The protocol driver 12 a,12 b may forward the control information 70 to the virtual networkinterface 14 directly or through a shared memory space. The controlinformation 70 includes a hash value 72 calculated using the hashfunction 20 or from the connection hash table 24. The controlinformation 70 may further include additional information for thepacket, such as data memory pointers addressing the packets or otherinformation. Protocol drivers 12 a, 12 b that do not include a hashfunction 20 or maintain a connection hash table 24 would not include ahash value 72 with their control information 70.

FIG. 4 illustrates operations performed by the protocol drivers 12 a, 12b and virtual network interface 14 to forward packets to an adapter 16a, 16 b, 16 c to transmit over the network. The protocol driver 12 a, 12b generates (at block 100) a packet having connection information in apacket header, which may comprise the source 52 and destination 54 portsand source 56 and destination 58 addresses. If (at block 102) there isnot one value in a first table (e.g., connection hash table 24) for theconnection information, then the protocol driver 12 a, 12 b applies (atblock 104) a function to the connection information to determine avalue. The function may comprise the hash function 20 to produce a hashvalue that directly or through further processing identifies one entryin the adapter table 22 that is associated with one adapter 16 a, 16 b,16 c. The determined hash value is added (at block 106) to the table,e.g., connection hash table 24, and associated with the connectioninformation. The added value is used to determine one of the adapters touse to transmit the packet.

If (at block 102) there is one value, e.g., hash value, for theconnection information in the table 24, then the protocol driver 12 a,12 b forwards (at block 108) the packet and the value in the connectionhash table 24 for the connection information to a network driver, suchas the virtual network interface 14. The protocol driver 12 a, 12 b mayinclude the value in the control information 70, such as in the hashvalue field 72 (FIG. 3) of the control information 70 sent with thepacket 50. At some point, a second table (e.g., the adapter table 22) isgenerated (at block 110). The adapter table 22 may be generated andmaintained by the virtual network interface 14. The value 72 sent withthe control information 70 for the packet 50 identifies an entry in thesecond table 22. Each entry in the second table (adapter table 22) isassociated with one of the network adapters 16 a, 16 b, 16 c. Using thevalue to determine one of the adapters may comprise determining theadapter 16 a, 16 b, 16 c associated with the entry in the second table22 identified by the value.

The network driver, e.g., virtual network interface 14, determines (atblock 112) whether the value used to determine one of the plurality ofadapters is received for the packet. The value 72 may be received withcontrol information 70 sent by the protocol driver 12 a, 12 b for thepacket 50. If so, the network driver 14 uses (at block 114) the valuereceived for the packet 50 to determine one of a plurality of adapters16 a, 16 b, 16 c to use to transmit the packet over a network.Otherwise, if a hash value 72 is not received for the packet 50, thenthe network driver 14 applies (at block 116) a function (e.g., hashfunction 26) to determine one (hash) value. In one embodiment, theprotocol driver, e.g., 12 b, may not include a hash function 20 togenerate a hash value to include with the control information 70 for thepacket 50. In such case, the virtual network interface 14 applies thehash function 26 to determine the hash value that identifies an entry inthe adapter table 22 associated with an adapter 16 a, 16 b, 16 c.

FIG. 5 illustrates operations performed by the virtual network interface14 to modify the assignment of the adapters to the entries in theadapter table 22 to perform load balancing for the adapters. Uponinitiating (at block 150) an operation to rebalance the adapter load,the virtual network interface 14 determines if the packet processingload at the adapters 16 a, 16 b, 16 c is unbalanced, i.e., the load atone adapter exceeds the load at the other adapters by a certainthreshold. If the load is not unbalanced, control ends. Otherwise, ifthe load is unbalanced, then the virtual network interface 14 generates(at block 154) a new second table 22 changing the assignment of entriesto the adapters 16 a. 16 b. 16 c. By changing the entries that theadapters are associated with, the hash values will then select differentadapters, thereby changing the adapter load.

In one embodiment, the hash function 20 processes information in thepacket header to produce a signature for the packet that is used as anindex into the adapter table 22, where the protocol driver 12 a, 12 b orvirtual network interface 14 may change the assignment of adapters 16 a,16 b, 16 c to entries in the table 22 to dynamically balance the packetload at the adapters. The hash function may comprise a Toeplitz hash orLevel 4 hash that is applied to certain of the information in the packetheader, such as the source and/or destination addresses and sourceand/or destination ports, i.e., a four tuple, or to just the source anddestination addresses. In one embodiment, a subset of the hash resultbits are used to provide an index into the adapter table 22. A mask maybe applied to the hash result to determine those bits indicating theindex (i.e., entry in the adapter table 22), where the maximum possiblevalue of the masked hash result indicates the number of entries in theadapter table 22.

ADDITIONAL EMBODIMENT DETAILS

The described embodiments may be implemented as a method, apparatus orarticle of manufacture using standard programming and/or engineeringtechniques to produce software, firmware, hardware, or any combinationthereof. The term “article of manufacture” as used herein refers to codeor logic implemented in hardware logic (e.g., an integrated circuitchip, Programmable Gate Array (PGA), Application Specific IntegratedCircuit (ASIC), etc.) or a computer readable medium, such as magneticstorage medium (e.g., hard disk drives, floppy disks,, tape, etc.),optical storage (CD-ROMs, optical disks, etc.), volatile andnon-volatile memory devices (e.g., EEPROMs, ROMs, PROMs, RAMs, DRAMs,SRAMs, firmware, programmable logic, etc.). Code in the computerreadable medium is accessed and executed by a processor. The code inwhich preferred embodiments are implemented may further be accessiblethrough a transmission media or from a file server over a network. Insuch cases, the article of manufacture in which the code is implementedmay comprise a transmission media, such as a network transmission line,wireless transmission media, signals propagating through space, radiowaves, infrared signals, etc. Thus, the “article of manufacture” maycomprise the medium in which the code is embodied. Additionally, the“article of manufacture” may comprise a combination of hardware andsoftware components in which the code is embodied, processed, andexecuted. Of course, those skilled in the art will recognize that manymodifications may be made to this configuration without departing fromthe scope of the embodiments, and that the article of manufacture maycomprise any information bearing medium known in the art.

The described operations may be performed by circuitry, where“circuitry” refers to either hardware or software or a combinationthereof. The circuitry for performing the operations of the describedembodiments may comprise a hardware device, such as an integratedcircuit chip, Programmable Gate Array (PGA), Application SpecificIntegrated Circuit (ASIC), etc. The circuitry may also comprise aprocessor component, such as an integrated circuit, and code in acomputer readable medium, such as memory, wherein the code is executedby the processor to perform the operations of the described embodiments.

The illustrated operations of FIGS. 4 and 5 show certain eventsoccurring in a certain order. In alternative embodiments, certainoperations may be performed in a different order, modified or removed.Moreover, steps may be added to the above described logic and stillconform to the described embodiments. Further, operations describedherein may occur sequentially or certain operations may be processed inparallel. Yet further, operations may be performed by a singleprocessing unit or by distributed processing units.

The foregoing description of various embodiments has been presented forthe purposes of illustration and description. It is not intended to beexhaustive or to limit the embodiments to the precise form disclosed.Many modifications and variations are possible in light of the aboveteaching.

1. A method, comprising: generating a packet, by a protocol driver,having connection information in a packet header; determining whetherthere is one value in a table for the connection information; forwardingthe packet and the determined value to a network driver in response todetermining that the table has one value for the connection information;and using, by the network driver, the forwarded value to determine oneof a plurality of adapters to use to transmit the packet over a network.2. The method of claim 1, wherein the value is included in controlinformation for the packet.
 3. The method of claim 1, furthercomprising: applying a function to the connection information todetermine a value in response to determining that the table does nothave one value for the connection information; and adding the value tothe table associated with the connection information, wherein the addedvalue is used to determine one of the adapters to use to transmit thepacket.
 4. The method of claim 1, further comprising: determining, bythe network driver, whether a one value used to determine one of theplurality of adapters is received for the packet, wherein the value isused to determine one of the plurality of adapters in response todetermining that the received packet includes one value; and applying afunction, by the network driver, to determine one value in response todetermining that one value is not received for the packet, wherein thedetermined value is used to determine one of the plurality of adaptersused to transmit the packet on the network.
 5. The method of claim 4,wherein there are a plurality of protocol drivers, including a firstprotocol driver that uses the table to determine one value and a secondprotocol driver that does not use one table, wherein one value is notforwarded for the packet received by the network driver from the secondprotocol driver.
 6. The method of claim 1, wherein the table comprises afirst table, further comprising: generating a second table, wherein thevalue identifies an entry in the second table, wherein each entry in thesecond table is associated with one of the network adapters, and whereinusing the value to determine one of the adapters comprises determiningthe adapter associated with the entry identified by the value.
 7. Themethod of claim 6, wherein the value comprises a hash value, furthercomprising: applying a hash function to the connection information forthe packet to determine the hash value associated with the connectioninformation; and adding one entry to the first table associating theconnection information with the hash value generated from the connectioninformation.
 8. The method of claim 6, further comprising: determiningwhether a load at the adapters is unbalanced; and generating a newsecond table changing the assignment of adapters to entries to rebalancethe load on the adapters.
 9. The method of claim 1, wherein theconnection information includes at least one of a destination address,source address, destination port, and source port.
 10. A system,comprising: (a) a plurality of network adapters; (b) a computer readablemedium including a table; (c) a processor coupled to the computerreadable medium; (d) a protocol driver comprising code executed by aprocessor to perform: (i) generating a packet having connectioninformation in a packet header; (ii) determining whether there is onevalue in the table for the connection information; (ii) forwarding thepacket and the determined value in response to determining that thetable has one value for the connection information; and (e) a networkdriver comprising code executed by a processor to perform: (i) receivingthe packet and the value forwarded from the protocol driver; (ii) usingthe forwarded value to determine one of a plurality of adapters to useto transmit the packet over a network.
 11. The system of claim 10,wherein the value is included in control information for the packet. 12.The system of claim 10, wherein the protocol driver code is furtherexecuted by the processor to perform: applying a function to theconnection information to determine a value in response to determiningthat the table does not have one value for the connection information;and adding the value to the table associated with the connectioninformation, wherein the added value is used to determine one of theadapters to use to transmit the packet.
 13. The system of claim 10,wherein the network driver code is further executed by the processor toperform: determining whether one value used to determine one of theplurality of adapters is received for the packet, wherein the value isused to determine one of the plurality of adapters in response todetermining that the received packet includes one value; and applying afunction to determine one value in response to determining that onevalue is not received for the packet, wherein the determined value isused to determine one of the plurality of adapters used to transmit thepacket on the network.
 14. The system of claim 13, wherein there are aplurality of protocol drivers, including a first protocol driver thatuses the table to determine one value and a second protocol driver thatdoes not use one table, wherein one value is not received for the packetreceived the from second protocol driver.
 15. The system of claim 10,wherein the table comprises a first table, wherein the network drivercode is further executed by the processor to perform: generating asecond table, wherein the value identifies an entry in the second table,wherein each entry in the second table is associated with one of thenetwork adapters, and wherein using the value to determine one of theadapters comprises determining the adapter associated with the entryidentified by the value.
 16. The system of claim 15, wherein the valuecomprises a hash value, wherein the protocol driver code is furtherexecuted by the processor to perform: applying a hash function to theconnection information for the packet to determine the hash valueassociated with the connection information; and adding one entry to thefirst table associating the connection information with the hash valuegenerated from the connection information.
 17. The system of claim 16,wherein the network driver code is further executed by the processor toperform: determining whether a load at the adapters is unbalanced; andgenerating a new second table changing the assignment of adapters toentries to rebalance the load on the adapters.
 18. The system of claim10, wherein the connection information includes at least one of adestination address, source address, destination port, and source port.19. The system of claim 10, wherein the network adapter includes a TOEengine and wherein the protocol driver comprises a TOE driver.
 20. Asystem, comprising: (a) a motherboard having slots for expansion cards;(b) a plurality of network adapters implemented on expansion cardscoupled to the motherboard;; (c) a computer readable medium including atable; (d) a processor coupled to the computer readable medium; (e) aprotocol driver comprising code executed by a processor to perform: (i)generating a packet having connection information in a packet header;(ii) determining whether there is one value in the table for theconnection information; (ii) forwarding the packet and the determinedvalue in response to determining that the table has one value for theconnection information; and (f) a network driver comprising codeexecuted by a processor to perform: (i) receiving the packet and thevalue forwarded from the protocol driver; (ii) using the forwarded valueto determine one of a plurality of adapters to use to transmit thepacket over a network.
 21. The system of claim 20, wherein the tablecomprises a first table, wherein the network driver code is furtherexecuted by the processor to perform: generating a second table, whereinthe value identifies an entry in the second table, wherein each entry inthe second table is associated with one of the network adapters, andwherein using the value to determine one of the adapters comprisesdetermining the adapter associated with the entry identified by thevalue
 22. An article of manufacture having stored thereon instructionsfor a protocol driver and a network driver coupled to a plurality ofnetwork adapters, the instructions when executed by a machine, causeoperations to be performed by the protocol driver and the networkdriver, the operations comprising: generating a packet, by the protocoldriver, having connection information in a packet header; determining,by the protocol driver, whether there is one value in a table for theconnection information; forwarding, by the protocol driver, the packetand the determined value to a network driver in response to determiningthat the table has one value for the connection information; and using,by the network driver, the forwarded value to determine one of theadapters to use to transmit the packet over a network.
 23. The articleof manufacture of claim 22, wherein the value is included in controlinformation for the packet.
 24. The article of manufacture of claim 22,wherein the operations further comprise: applying, by the protocoldriver, a function to the connection information to determine a value inresponse to determining that the table does not have one value for theconnection information; and adding, by the protocol driver, the value tothe table associated with the connection information, wherein the addedvalue is used to determine one of the adapters to use to transmit thepacket.
 25. The article of manufacture of claim 22, wherein theoperations further comprise: determining, by the network driver, whethera one value used to determine one of the plurality of adapters isreceived for the packet, wherein the value is used to determine one ofthe plurality of adapters in response to determining that the receivedpacket includes one value; and applying a function, by the networkdriver, to determine one value in response to determining that one valueis not received for the packet, wherein the determined value is used todetermine one of the plurality of adapters used to transmit the packeton the network.
 26. The article of manufacture of claim 25, whereinarticle of manufacture includes instructions for a plurality of protocoldrivers, including a first protocol driver that uses the table todetermine one value and a second protocol driver that does not use onetable, wherein one value is not forwarded for the packet received by thenetwork driver from the second protocol driver.
 27. The article ofmanufacture of claim 22, wherein the table comprises a first table, andwherein the operations further comprise: generating, by the networkdriver, a second table, wherein the value identifies an entry in thesecond table, wherein each entry in the second table is associated withone of the network adapters, and wherein using the value to determineone of the adapters comprises determining the adapter associated withthe entry identified by the value.
 28. The article of manufacture ofclaim 27, wherein the value comprises a hash value, wherein theoperations further comprise: applying, by the protocol driver, a hashfunction to the connection information for the packet to determine thehash value associated with the connection information; and adding, bythe protocol driver, one entry to the first table associating theconnection information with the hash value generated from the connectioninformation.
 29. The article of manufacture of claim 27, wherein theoperations further comprise: determining, by the network adapter,whether a load at the adapters is unbalanced; and generating, by thenetwork adapter, a new second table changing the assignment of adaptersto entries to rebalance the load on the adapters.
 30. The article ofmanufacture of claim 22, wherein the connection information includes atleast one of a destination address, source address, destination port,and source port.